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INTRODUCTION 


Much has been written about requirements in the Every Student Succeeds Act (ESSA) related to school 
classification, particularly with respect to approaches for determining if a school should be identified 
for support. However, less attention has been given to methods for determining when a school's 
performance is sufficient to exit such identification. Exit criteria offer clear goals to schools and districts 
that are connected to the state’s accountability system and reflect improvement expectations that 
should be sustained over time.',” If exit criteria are incoherently or unintentionally specified, they can 
lead to unattainable targets, inaccurate interpretations of whether schools need the same intensity of 
support, or distractions associated with managing perceptions instead of improvement.*Accordingly, 
the focus of this brief is to discuss principles to inform exit criteria and to provide guidance to help state 
leaders apply these principles. 


The essential question we attempt to address in this brief is, “How can we recognize sustainable 

and scalable school improvement when we see it?” Because this is a timely and important issue to 

state education leaders, we explicitly frame this topic in the context of school accountability systems 
developed in response to ESSA. In fact, we will focus most directly on approaches to inform exit 
standards for the categories of Comprehensive Support and Improvement (CSI) and Additional Targeted 
Support and Improvement (ATSI) given the explicit statutory requirements for these categories, detailed 
in the next section. We believe the principles addressed in this document have broad implications for 
accountability and support practices that transcend the policies and practices of ESSA. 


ESSA REQUIREMENTS 


Identification Criteria 


We will start by briefly reviewing the ESSA requirements related to school classification. The 
requirements for each of the three federally-required classification categories are summarized in 
Table 1*. The state may designate additional classifications (e.g. letter grades, reward schools) 
which may be integrated with or separate from those required by ESSA. 


1 Carroll, K. (2018). Improving Outcomes for Students: Bringing Accountability and School Improvement 
Together to Drive Change. CCSSO: Washington, D.C. 


2 For more information on sustaining successful school improvement efforts see: Council of Chief State School 
Officers and EducationCounsel. 2018. Deep Dive into Principle #10 of the CCSSO Principles of Effective School 
Improvement Systems. CCSSO: Washington, DC. 


3 D'Brot, Keng, & Landl (2018) Accountability Identification is only the Beginning: Monitoring and Evaluating 
Accountability Results and Implementation. CCSSO: Washington, D.C. 


4 For more information about understanding and implementing ESSA classifications see: 
Lyons, D'Brot, & Landl (2017). State Systems of Identification and Support under ESSA: A Focus on Designing and 
Revising Systems of School Identification. CCSSO: Washington, D.C. 


D’Brot, Lyons, & Landl (2017). State Systems of Identification and Support under ESSA: Evaluating Identification 
Methods and Results in an Accountability System. CCSSO: Washington, D.C. 


D'Brot, Keng, & Landl (2018) Accountability Identification is only the Beginning: Monitoring and Evaluating 
Accountability Results and Implementation. CCSSO: Washington, D.C. 


Table 1. Summary of ESSA Classification Requirements 


Category Which Schools? 


Comprehensive No fewer than the lowest performing 5% of all Title At least once every 
Support and | schools based on the state's system of meaningful three years 
Improvement (CSI) | differentiation with all indicators 


Title | schools that have failed to exit ATSI within a 
state-determined number of years 


All public high schools (not just Title |) with a 
graduation rate less than 67% 


Targeted Support | All schools with student groups that are consistently Annually 
and Improvement | underperforming, as defined by the state, based on 
(TSI) the state’s system of meaningful differentiation with all 

indicators 


Additional Any school in which the performance of any student _| State Determined* 
Targeted Support | group, on its own, would lead to identification for CSI 
and Improvement | because the student group performance falls within 
(ATSI) the range of the lowest performing five percent of 
all Title | schools (may or may not be a subset of TSI 
schools) 


*ESSA does not explicitly state how often ATSI schools must be identified. States have adopted different 
approaches in their ESSA state plans. 


Exit Criteria 


ESSA provides flexibility for states as they develop their statewide exit criteria. The law requires 
states to develop exit criteria for CSI and ATS! schools that must “ensure continued progress to 
improve student academic achievement and school success” (ESEA section 1111(d)(3)). 


With respect to CSI, the state must specify the criteria and the number of years by which 

the criteria must be satisfied. The time permitted to meet the exit criteria is determined by 

the state but may not exceed a maximum of four (4) years. If a school fails to exit CSI in the 
established timeframe this “shall, result in more rigorous state-determined action, such as the 
implementation of interventions (which may include addressing school-level operations.)” (ESEA 
section 1111(d)(3)(A)(i)(I)). 


Regarding ATSI, there only requirement is to develop exit criteria which, if not satisfied in a state- 
determined number of years, results in CSI identification. ESSA does not specify a maximum 
number of years. 
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GUIDING PRINCIPLES 


There is no single correct method for establishing appropriate exit standards. As with most 
decisions about the state’s accountability system, it is critical to align the goals, design decisions, 
and methods with the state’s policy priorities and theory of action for school improvement. For 
example, states may differ in the way they emphasize outcomes associated with post-secondary 
readiness or emphasize the balance between academic growth and proficiency. Consider also that 
states differ with respect to their long-term and interim goals and may sensibly choose to reflect 
these goals in the exit standards, which would lead to appropriate and defensible variation among 
criteria. Finally, states may differ with respect to improvement strategies and capacity, which 
certainly influence the decisions about which and how many schools can be supported and to what 
degree. Therefore, the principles and illustrations in the subsequent sections are not intended to 
be interpreted as uniform or overly proscriptive. Rather, we suggest states consider how these 
principles can be applied in context to complement established policy priorities. 


Five Principles for Exit Criteria 


We propose five principles to guide creation and evaluation of criteria for determining if schools 
have demonstrated sufficient and sustained progress. These principles and the central question 
they address are summarized in Table 2. 


Table 2. Five Principles to Inform Exit Criteria 


da tatell el (=) Essential Question 


Coherence Has the school addressed the factors that led to initial identification? 


: Has the improvement been sustained over time or coherently across 
congistency indicators? 


Significance Is the amount of progress exhibited meaningful? 


Is there evidence that the school benefitted from supports that are 
logically related to the area(s) of need? 


External Corroboration Are there multiple sources of evidence to validate the outcome? 


The first four principles: coherence, consistency, significance, and internal corroboration can 


Internal Corroboration 


directly inform the process of identifying and applying exit criteria. We suggest applying the fifth 
principle, external corroboration, as part of an ongoing monitoring and evaluation process. We will 
discuss these principles in more detail in the following section. 


Coherence 


Schools exhibiting meaningful improvement have addressed the factors that led to initial 
classification. For example, if a school was identified for improvement based on low rates of 
proficiency in mathematics, then it would be incoherent for that school to base exit classification for 
making progress in English language arts. 


This may be more straightforward to implement when decisions are made based on a performance 
profile. For example, in a system that renders an overall decision based on target performance in 
multiple categories, a coherent criterion is one that requires the school to address that category 

or categories that led to initial identification. In other words, to require that schools make sufficient 
improvement on the indicators or data elements that led to their identification. 


However, many state ESSA systems are based on performance indices that are compensatory. 

For example, in a compensatory system, multiple categories receive numerical scores which are 
combined via a weighted composite to produce an overall score. In such a system, a criterion 

that is based on an overall score increase may not address the factor(s) that led to initial 

placement. Such systems may permit exceptionally high performance in one area to offset very 

low performance in another. For this reason, states that use a compensatory system may wish 

to consider adding some requirements to define the acceptable performance profile(s) that will 
qualify for exit in order to ensure the critical and coherent areas of improvement are addressed. We 
discuss this more in a subsequent section of this brief. 


Consistency 


Claims of improvement are best supported when there is consistent evidence over time and across 
indicators. This is especially important if the performance in question is based on a small n-size, 


which can result in unexpected variances across years. 


This criterion may be especially important with respect to evaluating student group performance 
for ATSI exit. As noted previously, ESSA requires states to determine a timeframe to classify 

a school as CSI if the school fails to exit ATSI. The implication is that multiple years of low 
performance provides stronger evidence that the school is not serving that group of students 
effectively. It stands to reason that the decision to exit a school from support should also be 
grounded in sustained or consistent evidence of improvement over multiple years. 


To bolster evidence of consistency, especially when n-sizes are low or patterns are uneven, states 
might consider strategies such as: (1) setting a minimum threshold for multiple indicators or (2) 
evaluating progress based on a multi-year composite potentially using confidence intervals. 
Examining evidence for this criterion will differ depending on how exit criteria are operationalized. 
In the case of the first strategy, setting criteria for multiple indicators can help states better 
understand whether improvement across indicators results in meaningful change over time. 
Alternatively, setting indicator-specific criteria that spans over time can help states better 
understand whether improvement can generalize to other indicators in identified schools. 


Significance 


Especially in a system in which classifications are made on a normative basis (e.g. the lowest 5% of 
schools are classified) small differences that may or may not be meaningful can be the difference 
between schools that are classified as needing improvement and those that are not. Furthermore, 
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identified schools that demonstrate small but meaningful improvement may still fall below the 
identification threshold other non-identified schools also make improvements. While small 
differences may also influence classification in a criterion-referenced system, we draw a distinction 
because an assumed ‘meaningfulness’ built into the criteria. When it comes to determining exit 
criteria it is particularly important to determine if improvement is consequential or significant, and 
to sufficiently communicate expectations for improvement to schools to ensure they are well- 
informed of the progress they need to make in order to exit. 


This principle requires evidence that the magnitude and scope of improvement is more likely to 
be authentic than a statistical anomaly. Some of the strategies discussed previously can provide 
supporting evidence such as specifying performance profiles that take into account criteria for 
multiple indicators or requiring sustained improvement over time. Additionally, specific examples 
of how to design exit criteria are provided in other resources provided by CCSSO*. 


Another strategy to signal significance is to couple exit criteria with the state’s goals. For example, 
an exit criterion might specify that the school has met the state’s Measure of Interim Progress 
(MIP) toward the long-term goals for all students and all student groups. Another approach is 

to base expectations on research that provides more confidence that problematic patterns are 
unlikely to emerge. For instance, the state may analyze historical performance patterns and 
identify performance criteria where schools would have a very low probability of falling below the 
identification threshold in future years based on typical school conditions or performance alone. 


Internal Corroboration 


Internal corroboration refers to evidence that the school benefitted from practices to improve 
teaching and learning in areas logically related to the identified area(s) of need. Simply stated, if 
one assumes that certain practices lead to improved outcomes, do we know that those practices 
were implemented? If not, then improvement is less likely to be substantiated, capacity is less 
likely to have been increased, or assumptions within the theory of action® were not present or 
confirmed. Underlying this principle is the notion that schools that exit identification are unlikely to 
be re-identified because they should be able to sustain behaviors and practices that will result in 


sustained improvements. 


This principle of internal corroboration is likely to be more qualitative than quantitative and will 
be supported by evidence that inputs or interventions deemed central to the state’s model for 


5 For additional information on developing exit criteria, see: 


D’Brot, Lyons, & Landl (2017). State Systems of Identification and Support under ESSA: Evaluating Identification 
Methods and Results in an Accountability System. CCSSO: Washington, D.C.; Pages 13-15. 


School and District Improvement Frequently Asked Questions. Topic 1: Identification of Schools. 


Carrol, K (2018). Improving Outcomes for Students: Bringing Accountability and School Improvement Together to 
Drive Change. CCSSO: Washington, D.C.: Pages 17-18. 


6 For accountability systems, a theory of action refers to how the accountability system is intended to function 
in order to bring about the state’s desired outcomes, which should be tested and revisited through system 
implementation, monitoring, and evaluation (D’Brot, Keng, & Landl, 2018). 


school improvement were implemented with fidelity. For example, was a school improvement plan 
in place? Can the state verify that the plan met the state’s standards for quality and completeness 
and that the school, in fact, received the number, type, and quality of interventions and supports 
specified in that plan? Is there evidence of increased school capacity to engage in needs 
assessments in support of the school improvement plan? To the extent the school can provide 
and the state can substantiate affirmative responses to these questions, internal corroboration is 
strengthened. 


External Corroboration 


Although this may be the most ambitious principle to support, ideally, claims of authentic and 
sustainable improvement are bolstered by evidence that goes beyond what the accountability 
system and support system addresses. 


A plan to externally corroborate findings includes data from multiple sources that support claims 
of improvement. This may include more proximal indicators like: rising performance on a range of 
academic assessments in multiple content areas, increased quality and quantity of course offerings, 
favorable results from external accreditation or reviews, or evidence that teachers and leaders are 
highly qualified and meet the standards for professional practices. More distal indicators such as 
reductions in drop-out rates or improvements in college-going rates also lend strong support to a 
claim of real and sustained school improvement. 


In fact, a program of external corroboration may be more properly associated with an ongoing 
evaluation plan for the accountability system, rather than a factor that directly influences exit 
criteria. External corroboration represents an important source of evidence to help states monitor 
accountability system efficacy and improvement. 


APPLICATION 


In this section we start by listing several potential criteria for exit standards and show 

how they corresponded with the first four guiding principles. Then, we will provide some 
illustrative examples to show how exit standards could be designed and applied for schools 
with selected characteristics. 


Potential Criteria 


While far from exhaustive, a list of potential criteria for exit standards for CSI, ATSI, or another 
performance category are summarized in Table 3. 


swaysks Ayjiqe,unosoV/ 8181S VSSF 4Of CUE UD YX WIOf{U] O} SBUI/apIN“C) ‘quaWanocsduu JO aIUapIAW 


N 


Evidence of Improvement: Guidelines to Inform Exit Criteria for ESSA State Accountability Systems 


fee) 


Table 3. Potential Criteria for Exit Standards 


Internal 


Coherence  Consistenc Tre falnirersarer:) x 
y g (@foy age) elo) e- ale) a) 


Establish an overall performance threshold 
(with similar or different profiles/weights) 


Illustrative Examples 


One strategy for developing or evaluating possible exit standards is to consider how the collection 
of criteria satisfy each of the four principles. That is, states would select a combination of exit 
criteria that work together to address each principle. Illustrative examples are shown in Table 4. 
We have also included references to criteria that may benefit from additional information or clear 
definitions which would provide schools with transparent guidance on the progress needed to 
meet the state-wide exit criteria. 


Table 4. Illustrative Profiles for Exit Standards 


Context Illustrative Criteria 


School designated CSI; | School must achieve qualifying exit score associated with meaningful 

system identifies Title progress (as defined by X) toward long term goals on the overall 

| schools which rank in performance index for two (or some other state-defined number) consecutive 

the lowest 5% on overall | years AND the school must meet a minimum threshold score (as defined by 

index score X) for each indicator and each student group in the candidate year of exit. 
Qualifying improvement plan is in place and implementation of supports in 
all priority areas is substantiated (as defined by X). 


Schools failing to exit CSI after four years receive more rigorous state 
intervention (which would include example interventions for local 
communication). 


High school designated Graduation rate meets the exit standard associated with meaningful 
CSI for low graduation rate | progress (as defined by X) toward long term goals for all students and all 


student groups for two (or some other state-defined number) consecutive 
years. The school is not otherwise identified for CSI, TSI, or ATSI. Qualifying 
improvement plan is in place and implementation of supports in all priority 
areas is substantiated (as defined by X). 


Schools failing to exit CS! after four years receive more rigorous state 
intervention (which would include example interventions for local 
communication). 


School designated ATS! | The student group meets the exit standard for two consecutive years AND 
based on performance of | demonstrates progress at a rate sufficient to close the gap to the long-term 
a student group goal by 50% in six years or less (or some other state-defined timeline). 
The school is not otherwise identified for CSI, TSI, or ATSI. Qualifying 
improvement plan is in place and implementation of supports in all priority 
areas is substantiated. 


Schools failing to exit ATSI after three years are designated CSI. 


CONCLUSION’ 


As previously noted, there is no one correct method for establishing appropriate exit standards. 
Moreover, the specific context, policy priorities, and capacity of each state will certainly influence 
how exit standards are developed and implemented. Even so, we believe the principles discussed 
in this brief can serve as a useful guide to help state leaders develop and implement appropriate 
criteria to inform the conditions schools must meet to exit support classifications. 


7 Less focus was given in this brief to external corroboration, which is not meant to diminish its importance with 
respect to developing and validating exit criteria. On the contrary, efforts to ensure the criteria are meaningfully 
related to a range of prized short- and long-term outcomes are essential. We recommend that state leaders 
consider such activities as part of an overall plan to evaluate, monitor, and refine the system over time to ensure 
it appropriately supports the state’s policy priorities. Two recent and relevant resources that states leaders can 
review include the following: 


For confirming design decisions and building validity arguments for accountability systems: D’Brot, Keng, & Land 
(2018) Accountability Identification is only the Beginning: Monitoring and Evaluating Accountability Results and 
Implementation. CCSSO: Washington, D.C. 


For exit criteria and impact of system design: D’Brot, Lyons, & Land (2017). State Systems of Identification 
and Support under ESSA: Evaluating Identification Methods and Results in an Accountability System. CCSSO: 
Washington, D.C 
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